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WHICHEVER IS LONGER. FROM THE MAILING DATE OF THIS COMMUNICATION. 
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Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
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Status 

1 )S Responsive to communication(s) filed on 28 November 2005 , 
2a)KI This action is FINAL. 2b)n This action is non-final. 

3) n Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11. 453 O.G. 213. 

Disposition of Claims 

4) E Claim(s) 1-28 is/are pending in the application. 

4a) Of the above claim{s) is/are withdrawn from consideration. 

5) n Claim(s) is/are allowed. 

6) S Claim(s) 1-28 is/are rejected. 
?)□ Claim(s) is/are objected to. 

8) n Claim{s) are subject to restriction and/or election requirement 

Application Papers 

9) n The specification is objected to by the Examiner. 

10)S The drawing(s) filed on 28 November 2005 is/are: a)^ accepted or b)n objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 

Replacement drawing sheet(s) Including the correction is required if the drawing(s) is objected to. See 37 CFR 1 .121 (d). 
1 1 )□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 1 1 9 
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1 Certified copies of the priority documents have been received. 

2.n Certified copies of the priority documents have been received in Application No. . 

3.ISI Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 
Response to Amendment 

The amendment filed on 28 November 2005, has been entered and made of 
record. The replacement drawings filed 28 November 2005 has overcome the drawing 
objection made in the Non-Final office action filed 24 August 2005. Claims 1-28 are 
pending. 

Response to Arguments 

Applicant's arguments filed 28 November 2005 have been fully considered but 
they are not persuasive. 

Applicant argues "Evaluation of Artificial Reality" publication does not disclose 
the claimed generated nimbus. Examiner respectfully disagrees. Applicant provided 
support in the specification of the generated nimbus however, Applicant failed to 
distinguish how the claims overcome the prior art; 

Applicant argues "Evaluation of Artificial Reality" publication does not disclose 
lacking area for showing the observer a second actual object having a part which exists 
on the observer side than the actual object, and wherein said nimbus generating device 
also generates a nimbus image around a periphery of the lacking area. Examiner 
respectfully disagrees. Again, Applicant provided support in the specification of this 
lacking area however, Applicant failed to distinguish how the claims overcome the prior 
art. 
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Claim Objections 

Claim 1 is objected to because of the following informalities: On line 9, "an 
display" should be changed to "a display". Appropriate correction is required. 

Claim Rejections - 35 USC § 112 

The following is a quotation of the first paragraph of 35 U.S.C. 112: 

The specification shall contain a written description of the invention, and of the manner and process of 
making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the 
art to which it pertains, or with which it is most nearly connected, to make and use the same and shall 
set forth the best mode contemplated by the inventor of carrying out his invention. 

Claims 27 and 28 are rejected under 35 U.S.C. 112, first paragraph, as failing to 
comply with the enablement requirement. The claim(s) contains subject matter which 
was not described in the specification in such a way as to enable one skilled in the art to 
which it pertains, or with which it is most nearly connected, to make and/or use the 
invention. 

Referring to claims 27 and 28, measuring a velocity of the actual object is not 
supported by the specification. Even though paragraphs [0102-0104] mention speed of 
the humanoid robot, it is unclear from Applicant's disclosure how this velocity is 
measured. 

The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 
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Claims 27 and 28 recites the limitation "measures a velocity" in line 2. It is 
unclear how this velocity is measured. There is insufficient antecedent basis for this 
limitation in the claim. 

Claim Rejections • 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

1. Claims 1-8, 10-15, 17-22, and 24-26 rejected under 35 U.S.C. 103(a) as being 

unpatentable over "Evaluation of Artificial Reality: Chapter 1" (referred to EAR). 

EAR teaches the limitations of claims 1-8, 10-15, 17-22, and 24-28 with the 

exception of disclosing a nimbus (claims 1-4, 11, 12, 18, 19, 25) and a secorid object 

(claims 10, 17, 24, 26). However the system of EAR is used for superimposing an outer 

appearance fomied by CG on an assistive robot where the size of the displayed CG is 

increased larger than the actual robot to prohibit the robot from being seen behind the 

superimposed outer appearance. 

In regards to claims 1-4, 11, 12, 18, 19, 25- 

The configuration of an augmented reality image display system is shown in Fig. 
1 .32. Based on the visual point position and attitude information obtained from a three- 
dimensional position sensor attached to the head, CG are generated by a computer 
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(claims 3. 4. 12, 19, 25: an image generating device for generating computer 
graptiics in a figure corresponding to a figure of an actual object and a positional 
relationship between an observer, who is viewing the actual object, and the actual 
object) [pg. 10, lines 24-25; pg. 11. line 1]. Furthermore, the CG can be selected by 
taking the form of a character that the user likes, a deceased spouse, or the like (claims 

1, 2, 11, 18: an image selecting device for selecting computer graphics in a figure 
of an actual object and a positional relationship between an observer, who is 
viewing the actual object, and the actual object) [pg. 7, lines 1-3; Fig. 1 .3]. 
Simultaneously, a real image of the background is obtained from a CCD camera, and 
using the luminance key synthesis of a video mixer (claims 1-4, 11, 12, 18, 19: a 
combined image generating device for generating a combined image combining 
the graphics and the nimbus image), each are synthesized and displayed on an HMD 
as augmented reality (claims 1-4. 25: an image display processing device for 
displaying the combined image on an display which is viewed by the observer so 
that the combined image is superimposed on the actual object) [pg. 1 0, line 24 - 
pg. 1 1 , line 4]. The video mixer also doubles as the combined image sending device as 
disclosed in claims 2 and 4, where as shown in Fig. 1 .32. the video mixer sends the 
combined image of the CG and real image obtained by the camera to the HMD (claims 

2, 4: a combined image sending device for sending the combined image to the 
observer side). Furthermore, in this system, an lsotrak2 (made by Polhemus) is used 
as the magnetic three-dimensional position sensor. The computer is a TD-30 (made by 
Intergraph; CPU: Pentium 133 MHzx2, with built-in Open GL graphics accelerator), and 
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an FS5 (made by Virtual Research Systems) is used as the HMD [pg. 11, lines 5-10]. 
These provide the means as disclosed in claims 18 and 19. 

EAR does not explicitly teach a nimbus image, however, the system proposed in 
EAR is for superimposing an outer appearance formed by CG on an assistive robot 
performing tasks indoor. The error of the CG display position is such that the size of the 
displayed CG is increased larger than the actual robot [pg. 18, lines 7-15]. It would 
have been obvious for the system of EAR to have a nimbus image i.e., the error of the 
CG display position, around the periphery of the computer graphics in order to prohibit 
the robot from being seen behind the superimposed CG (claims 1-4, 11, 12, 18, 19, 25: 
a nimbus generating device for generating a nimbus image around a periphery of 
the computer graphics) . 

In regards to claims 5, 13, 20 - 

The rationale of claims 1-4, 1 1 , 12, 18 and 19 are incorporated herein. The system 
is for superimposing an outer appearance formed by CG on an assistive robot 
perfomiing tasks indoor. Since the total width of the robot performing a carrying task or 
the like is about 500 mm, and moreover, the robot is used indoors, it is assumed that it 
operates nearly 2 m in front of the user. The error of the CG display position is such 
that the size of the displayed CG is increased larger than the actual robot by as much 
as this error [pg. 18, lines 7-15]. This increased size of the display CG corresponds to 
the nimbus as disclosed. The maximum position error corresponds to the positional 
error when the robot is some distance away. The CG can be increased to 


Application/Control Number: 10/502,468 Page 7 

Art Unit: 2672 

counterbalance this error (claims 5, 13, 20: wherein said nimbus generating device 
estimates an error in measurement of a distance between the observer and the 
actual object, and generates based on the error the nimbus image having an 
adequate thiclcness for preventing the actuai object from being seen protruding 
from a periphery of the computer graphics) [pg. 1 8, lines 1 9 - 22]. 

In regards to claims 6, 14, 21 - 

The rationale of claims 1-4, 11, 12, 18 and 19 are incorporated herein. From the 
example cited in claim rejections 5/1 , 5/3, the maximum error (22 mm) corresponds to 
the positional error (44 mm) when the robot is 2 m away from the user. The resulting 
error is about one-tenth the size of the robot, and the CG can be increased to 
counterbalance this error. By correcting the three-dimensional position sensor, it is 
thought that the error can be controlled within a substantially permissible range (claims 
6, 14, 21: wherein said nimbus generating device generates the nimbus image 
having an adequate thicl(ness for not showing the observer a displacement which 
occurs between the actual and the computer graphics when the actual object or 
the observer moves) [pg. 18, lines 16-20; pg. 19, line 1]. 

In regards to claims 7, 15, 22 - 

The rationale of claims 1-4, 1 1 , 12, 18 and 19 are incorporated herein. In order to 
superimpose the CG on the robot and synthesize an image without a feeling of 
incompatibility, accuracy was required of the three-dimensional position sensor [pg. 12, 
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lines 1 - 3]. As seen in Fig. 1 .34, the rotation of the head to the right and left is 
measure by the sensor as a change in angle of direction, the tilting back and forth is 
measured as a change in the angle or elevation, and the tilting to right and left is 
measured as a change in the angle of torsion (claims 7, 15, 22: a detecting device for 
detecting at least one of distance and an angle between a display with is viewed 
by the observer and the actual object and a direction of the actual object seen 
from the observer) [pg . 1 3, lines 4-8]. 

In regards to claim 8 - 

The rationales of claims 1-4 are incorporated herein. The CG of a human shape on 
an arm-type care-giving robot used for a carrying task actually located at a position 2m 
away from the user [pg. 20, lines 7 - 9]. Furthermore, although a CG was 
superimposed on a target robot standing still, the robot actually changes position and 
aftitude over time (claim 8: wherein the actual object is a humanoid robot or an 
animal-type robot other than a human, which can move freely) [pg. 22, lines 7- 9]. 

In regards to claims 10, 17, 24, 26 - 

The rationale of claims 1-4, 11, 12, 18, 19 and 25 are incorporated herein. 
"Evaluation of Artificial Reality: Chapter 1" teaches the limitations of claims 10, 17, 24, 
26 except disclosing a second actual object. However, EAR teaches the CG of a 
human shape is superimposed on an arm-type care-giving robot used for a carrying 
task [pg. 20, lines 7 - 9]. The camera as shown in Fig. 1.32 will record the robot 
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carrying the item related to the carrying tasl^ and the video mixer superimposes the CG 
of a human shape on top of the robot. Since the CG is only of the human shape and 
does not contain a CG of the item related to the carrying task, it would have been 
obvious to one of ordinary skill in the art that the work station shown in Fig. 1 .32 
provides a mask to reveal the item related to the carrying task so that the super- 
imposed image sent to the HMD via the video mixer displays the item to the user so the 
robot looks like a character the user finds comforting performing the task (claims 10, 17, 
24, 26: wherein the computer graphics have a lacking area for showing the 
observer a second actual object having a part which exists on the observer side 
than the actual object). Although a CG was superimposed on a target robot standing 
still, the robot actually changes position and attitude over time [pg. 22, lines 7- 9]. The 
error of the CG display position is such that the size of the displayed CG is increased 
larger than the actual robot [pg. 18, lines 7-15]. This error, i.e. nimbus, compensates 
for the possible movement of the robot so when the robot moves, so that there is no 
chance that the actual robot will be seen apart from the CG superimposed. Thus, if the 
robot were carrying an item, it would have been obvious to include a nimbus around the 
item for the same reason (claims 10, 17, 24, 26: wherein said nimbus generating 
device also generates a nimbus image around a periphery of the lacking area). 

2. Claims 9, 16, 23 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
"Evaluation of Artificial Reality: Chapter 1" in view of Yuasa et al. (US Patent No. 
6,184,888 61). 
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"Evaluation of Artificial Reality: Chapter 1" (referred to as EAR) teaches the 
limitations of claims 9, 16, and 23 with the exception of disclosing receiving the 
computer graphics from outside via a network. However. Yuasa et al. teaches an image 
rendering apparatus and method intended for use in an environment in which the data 
transfer rate is limited, as in a network such as the Internet. 

EAR discloses a system for superimposing an outer appearance formed by CG on 
an assistive robot performing tasks indoor. The error of the CG display position is such 
that the size of the displayed CG is increased larger than the actual robot [pg. 18, lines 
7-15]. This increased size of the display CG corresponds to the nimbus as disclosed. 
The configuration of an augmented reality image display system is shown in Fig. 1.32. 
Based on the visual point position and attitude information obtained from a three- 
dimensional position sensor attached to the head, CG is generated by a computer [pg. 
10, line 24 - pg. 11, line 1], Furthermore, the CG can be selected by taking the form of 
a character that the user likes, a deceased spouse, or the like [pg. 7, lines 1-3; Fig. 1.3]. 
Simultaneously, a real image of the background is obtained from a CCD camera, and 
using the luminance key synthesis of a video mixer, each are synthesized and displayed 
on an HMD as augmented reality [pg. 10, line 24 -pg. 1 1 , line 4]. The video mixer also 
doubles as the combined image sending device, where as shown in Fig. 1.32, the video 
mixer sends the combined image of the CG and real image obtained by the camera to 
the HMD. Furthermore, in this system, an lsotrak2 (made by Polhemus) is used as the 
magnetic three-dimensional position sensor. The computer is a TD-30 (made by 
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Intergraph; CPU: Pentium 133 MHzx2, with built-in Open GL graphics accelerator), and 
an FS5 (made by Virtual Research Systems) is used as the HMD [pg, 11, lines 5-10]. 

Yuasa et al. teaches an image rendering apparatus and method intended for use in 
an environment in which the data transfer rate is limited, as in a network such as the 
Internet. As shown in Fig. 1, the rendering apparatus (100) resides in the computer 
(102). The rendering apparatus may form part of a graphics card located in the 
computer or may be a hardware accessory that plugs into or is othenA/ise connected to 
the graphics card of the computer [col. 6, lines 28-48]. The computer is shown 
connected via the I/O port (104) as a client of the network (106). The I/O port is 
connected directly or indirectly to the bus (108) to which other elements of the rendering 
apparatus and the computer are connected. The network, I/O port and bus constitute a 
path through which the rendering apparatus receives three-dimensional graphics data 
including texture data defining one or more textures (claims 9, 16, 23: comprising a 
communication device for receiving ttie computer graphics from outside via a 
network) [col. 6, lines 39-46]. 

Therefore, it would have been obvious to one of ordinary skill in the art at the time 
the invention was made to include the rendering apparatus of Yuasa et al. as a part of 
the Open GL graphics accelerator builtrin the computer disclosed in "Evaluation of 
Artificial Reality: Chapter 1", as well as incorporate the I/O port of Yuasa et al. to 
connect the computer disclosed in "Evaluation of Artificial Reality: Chapter 1" to a 
network because this would allow the user to import images from other sources, such 
as the internet, when the user is opting to select the CG by taking the form of a 
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character that the user likes, a deceased spouse, or the like ["Evaluation of Artificial 
Reality: Chapter 1": pg. 7, lines 1-3; Fig. 1.3]. Furthermore, importing the images from 
another source provides a larger range of characters the user may choose from as well 
as saving memory space on the computer by not having to generate and save 
multitudes of characters. 


Conclusion 

Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Offlce action. Accordingly, THIS ACTION IS MADE FINAL. . See MPEP 
§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Michelle K. Lay whose telephone number is (571) 272- 
7661 . The examiner can normally be reached on Monday - Friday, 7:00am - 4:30pm. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Michael Razavi can be reached on (571) 272-7664. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 

Michelle K. Lay 
Patent Examiner 
Art Unit 2672 , 
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